Liveness test method and apparatus

ABSTRACT

A liveness test method and apparatus is disclosed. A processor implemented liveness test method includes extracting an interest region of an object from a portion of the object in an input image, performing a liveness test on the object using a neural network model-based liveness test model, the liveness test model using image information of the interest region as provided first input image information to the liveness test model and determining liveness based at least on extracted texture information from the information of the interest region by the liveness test model, and indicating a result of the liveness test.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit under 35 USC §119(a) of KoreanPatent Application No. 10-2016-0106763 filed on Aug. 23, 2016, in theKorean Intellectual Property Office, the entire disclosure of which isincorporated herein by reference for all purposes.

BACKGROUND 1. Field

The following description relates to a method and apparatus detectingliveness of an object.

2. Description of Related Art

In a user recognition system, a computing apparatus may determinewhether to allow a user to have access to the computing apparatus basedon authentication information provided by the user. The authenticationinformation may include, for example, a password to be input by the userand biometric information of the user. The biometric information mayinclude, for example, information associated with a fingerprint, aniris, and a face of the user.

Recently, face anti-spoofing technology has been attracting a growinginterest as a security method for the user recognition system. A facespoof may be considered a type of an attack using an image, a video, ora mask, and thus identifying such an attack or counteracting theacceptance of the same as being genuine may be desirable. Accordingly, aface anti-spoofing technology may determine whether a face of a userinput to the computing apparatus is a fake face or a genuine face. Forthe determination, extraction of features, such as, for example, through‘hand-crafted’ analyses of local binary patterns (LBP), a histogram oforiented gradients (HoG), and a difference of Gaussians (DoG), may beperformed to determine whether the input face is fake or genuine. Suchfeature extraction analyses are considered ‘hand crafted’ because theyrequire each system to be manually designed for the correspondingcircumstance or implementation.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is the Summaryintended to be used as an aid in determining the scope of the claimedsubject matter.

In one general aspect, a processor implemented liveness test methodincludes extracting an interest region of an object from a portion ofthe object in an input image, performing a liveness test on the objectusing a neural network model-based liveness test model, the livenesstest model using provided image information of the interest region asfirst input image information to the liveness test model and determiningliveness based at least on extracted texture information from the imageinformation of the interest region by the liveness test model, andindicating a result of the liveness test.

The liveness test model may use image information of the input image orimage information of the portion of the object in the input image asprovided second input image information to the liveness test model thatis also provided the first input image information, and the determiningof the liveness may be based at least on the extracted textureinformation from the first input image information and spatialinformation extracted based on the second input image information by theliveness test model.

The method may further include adjusting a pixel size of at least theimage information of the portion of the object in the input image, afterthe extracting of the interest region, to generate the second inputimage information.

An effective resolution of the first input image information may behigher than an effective resolution of the second input imageinformation.

A pixel size of the first input image information may be equal to apixel size of the second input image information.

The liveness test model may include a recurrent connection structurethat is configured so a determination of liveness of the object, withrespect to the input image being a current frame, is dependent onresults of the liveness test model obtained with respect to a previousframe.

The first input image information and the second input image informationmay be input to the liveness test model through different input layersof the liveness test model.

The liveness test model may independently perform filtering on the firstinput image information and the second input image information beforeperforming further filtering on a combination of information of resultsfrom both the performed filtering on the first input image informationand the performed filtering on the second input image information.

The extracting of the interest region may include extracting a selectportion including a pupil region as the interest region from the portionof the object in the input image.

The liveness test model may use image information of the portion of theobject in the input image, within a facial region of the object andincluding the select portion, as provided second input image informationto the liveness test model that is also provided the first input imageinformation, and the determining of the liveness may be based at leaston the extracted texture information from the select portion and spatialinformation extracted based on the second input image information by theliveness test model.

The extracting of the interest region may include extracting a selectportion, from the portion of the object in the object in the inputimage, including two eyes or at least two of an eye, a nose, and lips asrespective interest regions.

The extracting of the interest region may include extracting respectiveselect portions, from the portion of the object in the input image,including at least one of an eye, a nose, or lips as the interestregion.

The liveness test model may use image information of the portion of theobject in the input image, within a facial region of the object andincluding one or more of the respective select portions, as providedsecond input image information to the liveness test model that is alsoprovided the first input image information, and the determining of theliveness may be based at least on extracted texture information from theone or more of the respective select portions by the liveness test modeland spatial information extracted based on the second input imageinformation by the liveness test model.

The extracting of the interest region may include extracting respectiveone or more select portions, from a fingerprint region as the portion ofthe object in the input image, as respective interest regions.

The liveness test model may use image information of the portion of theobject in the input image, as provided second input image information tothe liveness test model that is also provided the first input imageinformation, and the determining of the liveness may be based at leaston extracted texture information from the respective one or more selectportions by the liveness test model and spatial information extractedbased on the second input image information by the liveness test model.

The extracting of the interest region may include extracting a selectportion, from a vein region as the portion of the object in the inputimage, as the interest region.

The liveness test model may use image information of the portion of theobject in the input image, as provided second input image information tothe liveness test model that is also provided the first input imageinformation, and the determining of the liveness may be based at leaston the extracted texture information from the select portion and spatialinformation extracted based on the second input image information by theliveness test model.

The method may further include normalizing the input image, and theextracting of the interest region may include extracting the interestregion of the object from the normalized input image.

The extracting of the interest region may include detecting featurepoints in the input image to determine the portion of the object in theinput image, and determining the interest region based on the detectedfeature points.

A location of the interest region may be determined based on adetermined type of biometric information for which the liveness test isdetermined to be performed.

The method may further include selectively controlling access tofunctions of a computing apparatus based on results of the performedliveness test.

In one general aspect, there is provided a non-transitorycomputer-readable storage medium storing instructions, that whenexecuted by a processor, cause the processor to perform one or more orall operations described herein.

In one general aspect, a liveness test computing apparatus includes atleast one processor configured to extract an interest region of anobject from a portion of the object in an input image, and perform aliveness test on the object using the neural network model-basedliveness test model, the liveness test model using image information ofthe interest region as provided first input image information to theliveness test model and configured to determine liveness based at leaston extracted texture information from the image information of theinterest region by the liveness test model.

The liveness test computing apparatus may further include a memoryconnected to the processor, and the memory may store trained weightinginformation of the liveness test model.

The liveness test model may be configured to use image information ofthe input image or image information of the portion of the object in theinput image as provided second input image information to the livenesstest model that is also provided the first input image information, witha pixel size of at least the image information of the portion of theobject in the input image being adjusted after the extracting of theinterest region to generate the second input image information, theapparatus may be configured to provide a reference value indicating aresult of the liveness determination, and the determining of theliveness may be based at least on the extracted texture information fromthe first input image information and spatial information extractedbased on the second input image information by the liveness test model.

The liveness test computing apparatus may be further configured to inputthe first input image information and the second input image informationto the liveness test model through different configured input layers ofthe liveness test model.

The liveness test model may be configured to independently performfiltering on the first input image information for extracting thetexture information and filtering on the second input image informationfor extracting the spatial information before performing furtherfiltering on a combination of information of results from both theperformed filtering on the first input image information and performedfiltering on the second input image information.

The liveness test model may include a recurrent connection structurethat is configured so a determination of liveness of the object, withrespect to the input image being a current frame, is dependent onresults of the liveness test model obtained with respect to a previousframe.

The processor may be further configured to selectively controllingaccess to functions of the liveness test computing apparatus based onresults of the performed liveness test.

In a general aspect, a liveness test computing apparatus includes atleast one processor configured to extract an interest region of anobject within a detected face region in an input image, perform aliveness test on the object using a convolutional neural network byproviding a first input layer of the convolutional neural network imageinformation of the interest region as first input image information anddetermining liveness based at least on texture information extractedfrom the image information of the interest region through a firstconvolutional layer of the convolutional neural network, and selectivelycontrol access to other functions of the liveness test computingapparatus based on results of the performed liveness test.

The liveness test computing apparatus may be a mobile phone, tablet, orpersonal computer and the other functions include at least access toinformation stored in a memory of the liveness test computing apparatus.

The processor may be configured to provide a second input layer of theconvolutional neural network second input image informationcorresponding to the detected face region, with a pixel size of at leastimage information of the face region being adjusted after the extractingof the interest region to generate the second input image information,and the determining of the liveness may be based at least on theextracted texture information from the first input image information andspatial information extracted from the second input image informationthrough a second convolutional layer of the convolutional neural networkindependent of the first convolutional layer.

The convolutional neural network may further include a thirdconvolutional layer configured to perform filtering on a combination ofresults dependent on filtering by the first convolutional layer andresults dependent on filtering by the second convolutional layer.

Other features and aspects will be apparent from the following detaileddescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A through 1D are diagrams illustrating examples of livenesstests.

FIG. 2 is a flowchart illustrating an example of a liveness test method.

FIGS. 3A through 3D are diagrams illustrating examples of a method ofextracting an interest region.

FIG. 4A is a diagram illustrating an example of a liveness test method.

FIG. 4B is a diagram illustrating an example of a liveness test method.

FIGS. 5 and 6 are diagrams illustrating examples of a liveness testmethod.

FIG. 7 is a diagram illustrating an example of a liveness testapparatus.

FIG. 8 is a diagram illustrating an example of a computing apparatus.

Throughout the drawings and the detailed description, unless otherwisedescribed or provided, the same drawing reference numerals will beunderstood to refer to the same or like elements, features, andstructures. The drawings may not be to scale, and the relative size,proportions, and depiction of elements in the drawings may beexaggerated for clarity, illustration, and convenience.

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader ingaining a comprehensive understanding of the methods, apparatuses,and/or systems described herein. However, various changes,modifications, and equivalents of the methods, apparatuses, and/orsystems described herein will be apparent after an understanding of thedisclosure of this application. For example, the sequences of operationsdescribed herein are merely examples, and are not limited to those setforth herein, but may be changed as will be apparent after anunderstanding of the disclosure of this application, with the exceptionof operations necessarily occurring in a certain order. Also,descriptions of features that are known in the art may be omitted forincreased clarity and conciseness.

The features described herein may be embodied in different forms, andare not to be construed as being limited to the examples describedherein. Rather, the examples described herein have been provided merelyto illustrate some of the many possible ways of implementing themethods, apparatuses, and/or systems described herein that will beapparent after an understanding of the disclosure of this application.

The terminology used herein is for the purpose of describing particularexamples only, and is not to be used to limit the disclosure. As usedherein, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. As used herein, the term “and/or” includes any one and anycombination of any two or more of the associated listed items. As usedherein, the terms “include,” “comprise,” and “have” specify the presenceof stated features, numbers, operations, elements, components, and/orcombinations thereof, but do not preclude the presence or addition ofone or more other features, numbers, operations, elements, components,and/or combinations thereof.

In addition, terms such as first, second, A, B, (a), (b), and the likemay be used herein to describe components. Each of these terminologiesis not used to define an essence, order, or sequence of a correspondingcomponent but used merely to distinguish the corresponding componentfrom other component(s).

Throughout the specification, when an element, such as a layer, region,or substrate, is described as being “on,” “connected to,” or “coupledto” another element, it may be directly “on,” “connected to,” or“coupled to” the other element, or there may be one or more otherelements intervening therebetween. In contrast, when an element isdescribed as being “directly on,” “directly connected to,” or “directlycoupled to” another element, there can be no other elements interveningtherebetween. Likewise, expressions, for example, “between” and“immediately between” and “adjacent to” and “immediately adjacent to”may also be construed as described in the foregoing.

Unless otherwise defined, all terms, including technical and scientificterms, used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this disclosure pertainsconsistent with and after an understanding of the present disclosure.Terms, such as those defined in commonly used dictionaries, are to beinterpreted as having a meaning that is consistent with their meaning inthe context of the relevant art and the present disclosure, and are notto be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

Hereinafter, examples are described in detail with reference to theaccompanying drawings.

FIGS. 1A through 1D are diagrams illustrating examples of liveness test.

A liveness test, or used interchangeably with liveness detection, isused to detect whether an object is live or not, for example, whether aface captured using a camera is a genuine face or a fake face. The term“liveness” described herein is used to distinguish a living object froma lifeless object, for example, a photo or an image, and a replica usedfor fakery. Depending on embodiment, one or more liveness testsdescribed herein may be used to detect liveness of a target for whichuser authentication is to be performed in user login, mobile payment,access control, and the like. The liveness test may thus filter out, orblock, an attempt for the authentication, for example, a spoofing attackmade using a substitute, for example, an image, a video, a mask, and areplica, and thus prevent erroneous authentication. In addition, beingdata driven, the liveness test may be implemented more accurately and bemore readily available, such as in requiring less processing resources,than previous hand-crafted spoof prevention techniques.

FIG. 1A is a diagram illustrating an example of facial recognition beingperformed on a computing apparatus 120 according to one or moreembodiments. Referring to FIG. 1A, a liveness test apparatus is includedin the computing apparatus 120, or is represented by the computingapparatus 120. The computing apparatus 120 authenticates a user 110 whomakes an attempt for an access to the computing apparatus 120 throughfacial recognition. For example, the user 110 may make such an attemptfor user authentication in the computing apparatus 120 to unlock thecomputing apparatus 120, and the computing apparatus 120 may capture aface image of the user 110 using an image capturing device, for example,a camera 130, and analyze the obtained face image to allow or rejectauthenticating the user 110. The computing apparatus 120 may determinewhether to authenticate the user 110 by comparing the obtained faceimage to a preregistered image. The computing apparatus 120 may be, forexample, a smartphone, a wearable device, a tablet computer, amini-laptop (netbook), a laptop, a desktop, a personal digital assistant(PDA), a set-top box, a television (TV), a biometrics-based door lock, asecurity device, and a vehicle start device.

As illustrated in FIG. 1A, when the computing apparatus 120 operates ina lock mode, the computing apparatus 120 may authenticate the user 110through the facial recognition. For example, when the user 110 is anauthorized user, the user 110 may cancel the lock mode of the computingapparatus 120 through the facial recognition performed by the computingapparatus 120. The user 110 may register or store, in advance, a faceimage of the user 110 in the computing apparatus 120, and the computingapparatus 120 may store, in a storage or a cloud storage, a registeredface image of the user 110 obtained by capturing a face of the user 110.

To unlock the computing apparatus 120, the user 110 may capture a faceimage including at least a portion of the face of the user 110 throughthe computing apparatus 120. For another example, the user 110 mayalternatively be an unauthorized user and be attempting to unlock thecomputing apparatus 120 by causing erroneous authentication in thecomputing apparatus 120 using a spoofing technique. For example, theunauthorized user, may present, to the camera 130, an image on which aface of an authorized user is printed or a replica in a form of the faceof the authorized user in order to cause the erroneous authentication.Here, the liveness test apparatus may prevent the erroneousauthentication that may be caused by such a fakery and spoofingtechnique in previous anti-spoofing techniques. The liveness testapparatus may detect an attempt for authentication that is suspicious asfakery, and effectively prevent fakery-based erroneous authentication.

In one example, the liveness test apparatus may perform a liveness testusing information of a portion of an image obtained by the camera 130,which is considered important for the liveness test, in lieu of anentire region of the image. For example, as illustrated in FIG. 1, theliveness test apparatus may extract an interest region of the user'sbody (or a region of interest [ROI]), for example, an eye, a nose, andlips, as a portion of and from within a face region 140, and perform theliveness test on an object based on texture information of the extractedinterest region. The liveness test apparatus may use only theinformation of the extracted portion to reduce resources and an amountof calculation or computation by a liveness test model that implementsthe liveness test and detect liveness of the object more rapidly. Here,if the entire region of the image were extracted instead of the interestregion(s), the corresponding image data may be required to bedownsampled to a lower pixel resolution for liveness test analyses, orthe computations of the full entire region at full resolution may demandtoo much processing resources and could not be able to be implemented inreal time, compared to the above liveness test apparatus approachaccording to one or more embodiments that selectively considers suchinterest region(s), which may use less processing resources and thus maybe available for mobile devices, and which may be implemented in realtime. In addition, such entire region analyses approaches may result inlower accuracies than the above liveness test apparatus approachaccording to one or more embodiments, as the entire region analysesapproaches would only be able to consider spatial information due to thelower pixel resolution, while the above liveness test apparatus approachaccording to one or more embodiments may consider texture and spatialfeature information. Still further, such entire region analysesapproaches may require extensive training resources or be unable to besuccessfully trained, compared to the above liveness test approachaccording to one or more embodiments.

In another example, the liveness test apparatus may also perform theliveness test using information of a larger portion of the image oranother image obtained by the camera 130, and thus perform the livenesstest based on global shape information of the object indicated in theface region 140, in addition to the texture information of the interestregion. For example, the global shape information of the object may befocused on, or limited to, shape information of the body area from whichthe interest region is extracted, though embodiments are not limitedthereto. Thus, the liveness test apparatus may detect the liveness moreaccurately based on the texture information and the shape informationusing information from both the extracted interest region and a globalregion within the captured image.

In still another example, the liveness test apparatus may perform theliveness test based on a texture, a shape, and a movement characteristicof the object based on a lapse of time, which are indicated in the faceregion 140. The liveness test apparatus may extract at least oneinterest region from within the face region 140, and detect the livenessbased on texture information of the interest region, shape informationindicated in the face region 140, and movement information of the objectbased on a lapse of time. Based on such various types of information,the liveness test apparatus may detect whether the object is fake orgenuine more accurately. A difference may be present between a fakeobject and a live object in terms of a texture, a shape, and a movementbased on a lapse of time, and thus the liveness test apparatus maydetermine such a difference and determine liveness of an object moreaccurately using the determined difference.

For performing the liveness test on the object, the liveness testapparatus may use a neural network-based liveness test model, forexample. The liveness test model refers to a model that may output areference value that indicates the liveness of the object based on inputinformation. The liveness test model may intuitively determine theliveness of the object, such as by being pre-trained based on trainingdata, and thus data driven. The liveness test apparatus may determinewhether the object is a fake object or a live object based on thereference value output from the liveness test model. For example, theoutput reference value may be a probability value, indicating theprobability of the object being a fake or live object, or merely acorresponding binary classification indication, for example.

When the liveness test apparatus determines that the face of the user110 captured by the camera 130 is not a fake but a genuine one, thecomputing apparatus 120 may then perform further user authentication todetermine whether the user 110 is a registered user by comparing theobtained face image to a preregistered image, such as through use offurther facial recognition operations. When the user authentication issuccessful, the user 110 may successfully unlock the computing apparatus120. Conversely, when the user authentication is unsuccessful, the user110 may not unlock the computing apparatus 120 and the computingapparatus 120 may continuously operate in the lock mode. When theliveness test apparatus determines that the face of the user 110captured by the camera 130 is fake, the computing apparatus 120 may notproceed to the further user authentication and thus not perform thefacial recognition, and may rather reject an access by the user 110 tothe computing apparatus 120.

According to examples described herein, more accurately detectingliveness of an object may reduce a possibility of erroneousauthentication that may be caused by fakery. In addition, based on aresult of the detecting of the liveness, whether to perform further userauthentication including facial recognition may first be determined, andthus resources and power consumption typically needed for the furtheruser authentication may be reduced.

FIG. 1B is a diagram illustrating an example of iris recognition.Referring to FIG. 1B, the computing apparatus 120 authenticates the user110 who makes an attempt for an access to the computing apparatus 120through iris recognition. An iris region of the user 110 is selectivelycaptured or cropped by the camera 130 or an iris recognition sensor, andthe computing apparatus 120 allows or rejects authenticating the user110 by analyzing biometric information of the iris region in theobtained image using a liveness test model. As illustrated in FIG. 1B,the liveness test apparatus detects at least one eye region, forexample, an eye region 150 and an eye region 160, in the obtained image,extracts an interest region including the eye regions 150 and 160 andless than the whole face, and detects liveness of the user 110 based oninformation of the extracted interest region.

FIG. 1C is a diagram illustrating an example of fingerprint recognition.Referring to FIG. 1C, the computing apparatus 120 authenticates the user110 who makes an attempt for an access to the computing apparatus 120through fingerprint recognition. A fingerprint of the user 110 iscaptured through a camera or a fingerprint recognition sensor. Forexample, as illustrated in FIG. 1C, the fingerprint is captured when theuser 110 touches a display 125, captured through the fingerprintrecognition sensor embedded in a home button 170, or otherwise captured.The computing apparatus 120 determines whether to authenticate the user110 by analyzing biometric information of a fingerprint pattern in theobtained fingerprint image using a liveness test model. Here, theliveness test apparatus may extract an interest region in which acharacteristic of the fingerprint pattern is conspicuous from thefingerprint image, and detect liveness of the user 110 based oninformation of the extracted interest region which is less than thewhole fingerprint region.

FIG. 1D is a diagram illustrating an example of vein pattern recognition(VPR). Referring to FIG. 1D, a computing apparatus 180, which isprovided as a watch-type wearable device, authenticates the user 110 whomakes an attempt for an access to the computing apparatus 180 throughVPR. When the user 110 wears the computing apparatus 180 around a wristof the user 110, for example, and makes an attempt for authentication tohave an access to the computing apparatus 180 and/or another computingdevice in communication with the computing apparatus 180, a light sourceembedded in the computing apparatus 180 may radiate infrared lighttowards skin of the user 110 and the infrared light reflected byhemoglobin in a vein of the user 110 may be recorded in a charge-coupleddevice (CCD) camera embedded in the computing apparatus 180. Thecomputing apparatus 180 may determine whether to authenticate the user110 by analyzing biometric information of a vascular pattern indicatedin a vein image captured by the CCD camera using a liveness test model.Here, the liveness test apparatus may extract an interest region inwhich a characteristic of the vascular pattern is conspicuous from theobtained vein image, and detect liveness of the user 110 making theattempt for the authentication based on information of the extractedinterest region which is less than the whole vein image.

Hereinafter, a method of performing a liveness test on an object by aliveness test apparatus will be described in more detail with referenceto the accompanying drawings.

FIG. 2 is a flowchart illustrating an example of a liveness test method.

Referring to FIG. 2, in operation 210, a liveness test apparatusreceives an input image. The input image may include at least one of astill image or a video image including a plurality of image frames. Theinput image may be a color image, a black-and-white image, or aninfrared image, for example. The input image may include biometricinformation of, for example, a face, an iris, a fingerprint, and a veinof a user. For example, in iris recognition, an infrared image obtainedby capturing an iris region may be input to the liveness test apparatus.

In operation 220, the liveness test apparatus may normalize the inputimage. For example, after receiving the input image, the liveness testapparatus may rotate, scale up or down, or crop the input image. Theliveness test apparatus may also perform various image preprocessingmethods on the input image in the alternate or in addition to therotating, the scaling up or down, or the cropping. Such imagepreprocessing methods may include, for example, conversion of brightnessor color, and removal of an artifact or a blur. Depending on embodiment,operation 220 may or may not be included in the liveness test method, oroperation 220 may be selectively performed.

In operation 230, the liveness test apparatus extracts an interestregion of an object from the input image, or the normalized input image.A shape and a size of the interest region of the object to be extractedmay not be limited to a specific shape and size, but is a portion of theimage and representative of a portion of the imaged object. The interestregion refers to a region of the object that may be useful for theliveness test on the object based on a type of biometric information ortype of biometric authentication, and includes a feature determined orpredetermined to be significant in the determination of liveness of theobject. For example, in facial recognition, the liveness test apparatusmay extract, as an interest region, a select portion of the objectincluding at least one of an eye, a nose, and lips from a whole of theimaged face region of a user in the input image. In an example, theinterest region may be determined so as to respectively includesubstantially only the example one eye, the nose, or the lips, forexample. For the facial recognition, a nose region, an eye region, and alips region of the user may provide a valuable clue to determination ofliveness of the user, e.g., compared to other facial regions, when theliveness determination is made by a liveness test model of the livenesstest apparatus.

For another example, in iris recognition, the liveness test apparatusmay extract a portion including a pupil region as an interest regionfrom an eye region of a user in an input image. For example, theextracted interest region may be determined so as to includesubstantially only the pupil region of the eye region, for example. Forthe iris recognition, a corneal reflex appearing in the pupil region andimage information of a neighboring region of the corneal reflex mayprovide a clue to determination of liveness of the user.

In one example, the liveness test apparatus may detect feature points,such as, for example, facial landmarks, of a face in an input image, anddetermine an interest region based on the detected feature points. Theliveness test apparatus may detect a global region, e.g., within theinput image, based on the feature points, and extract the interestregion based on feature points detected within the global region. Theglobal region may be a region smaller than an entire region of the inputimage, but greater than the interest region. For example, in the facialrecognition, a face region from which a background region is excludedmay be detected as the global region. For example, a global image of theglobal region may be determined so as to substantially only include theface region, with an interest image of the interest region being aportion within the global image. For another example, in the irisrecognition, an eye region may be detected as the global region. The eyeregion may be determined so as to include substantially only one eye orboth eyes, for example. Here, other facial component regions including,for example, a nose, lips, and eyebrows, may be excluded from theexample global region. The liveness test apparatus may also determinethe interest region in the input image using a visually salient regiondetecting method. However, examples are not limited thereto.

For still another example, in fingerprint recognition, the liveness testapparatus may extract a portion from a fingerprint pattern including aselect fingerprint region as an interest region from an input imageincluding the fingerprint pattern. For example, the liveness testapparatus may extract, as the interest region, a select portion of thefingerprint region in which a change in the fingerprint pattern isdetermined conspicuous. For example, the liveness test apparatus mayextract, as the interest region, a portion in which feature points, forexample, an ending point of a ridge, a bifurcation point, a core point,and a delta point, are concentrated, from an entire fingerprint regionin the input image.

For yet another example, in VPR, the liveness test apparatus may extracta portion including a vein region as an interest region from an inputimage including a vein distribution pattern. For example, the livenesstest apparatus may extract, as the interest region, a portion includinga vein pattern of a wrist from an entire hand region or veindistribution pattern in the input image, or a portion of the veindistribution pattern in which one of an end and a bifurcation point of ablood vessel is intensively distributed in the vein distributionpattern.

In operation 240, the liveness test apparatus performs a liveness teston the object based on the interest region. The liveness test apparatusmay perform the liveness test on the object using a neural networkmodel-based liveness test model that uses image information of theinterest region as an input. The image information of the interestregion may thus include texture information as the pixel values of thepixels included in the interest region, for example, the respectivecolor values and brightness values. The liveness test model may bepre-trained through a supervised learning method and perform nonlinearmapping, and thus may perform desirably in distinguishing differences inthe liveness test.

In one example, the liveness test model may use the image information ofthe interest region as an input, and the liveness test apparatus maydetermine the liveness of the object based on an output value of theliveness test model. The liveness test apparatus may reduce resourceconsumption and an amount of calculation or computation by performingthe liveness test based on a partial interest region in lieu of anentire region of the input image. Thus, when trained, the liveness testmodel may output a reference value to indicate the liveness based onintuitively observed local texture information indicated in the interestregion.

In another example, the liveness test model may use the imageinformation of the interest region as a first input, and use imageinformation of the input image of which a size is adjusted as a secondinput. For example, the size or dimensions of image information of theinterest region may be maintained, such as by cropping the interestregion from the input image, and then the size or dimensions of theinput image may be adjusted to be the same as a size or dimensions ofthe interest region, which may result in a re-scaling or down-samplingof the original resolution of the input image. Thus, the effectiveresolution of the interest region may be greater (or higher) than theeffective resolution of the resized input image. Image information ofthe input image of which the size is adjusted may include shapeinformation that may be beneficial for the liveness test. The inputimage may correspond to the aforementioned global image, e.g., less thana whole captured image. Accordingly, compared to previous approaches,interest regions of a body that may provide liveness clues can be used,such as with a highest captured resolution, so texture indicatinginformation may be maintained and discernable by the liveness test modelfor each input image. In addition, though the whole input image or theglobal image may be down-scaled or downsampled, at least spatialfeatures can be considered from each input image. Thus, as demonstratedin the below example of FIG. 4B, by considering the interest region andthe input image or global image, both texture and spatial features maybe observed and considered by the liveness test model for a single inputimage. Likewise, as demonstrated in the below example of FIG. 4A, byconsidering the interest region with sufficient resolution, textures ofthe interest region may be considered by the liveness test model for asingle image. As the interest region has already been determined by theliveness test apparatus as a select region that provides a clueregarding liveness, the resolution of the select interest region may bemaintained for such a texture analysis of the select interest region,e.g., independent of select spatial or texture analyses that may or maynot be performed on the input image without the maintained resolution,thereby performing liveness test analyses on the input image withoutundue computational requirements. The liveness test model may output thereference value to indicate the liveness based on the local textureinformation indicated in the interest region and global shapeinformation indicated in the input image of which the size is adjusted.

The liveness test model may output, as the reference value, at least oneof a first probability value of the object in the input image being afake object and a second probability value of the object in the inputimage being a live object. The liveness test apparatus may determinewhether the object in the input image is live or fake based on thereference value. For example, when the first probability value isprovided as an output by the liveness test model, the liveness testapparatus may determine the object to be a fake object in response tothe first probability value being greater than a threshold value, and tobe a live object in response to the first probability value being lessthan or equal to the threshold value. For another example, when thesecond probability value is provided as an output by the liveness testmodel, the liveness test apparatus may determine the object to be a liveobject in response to the second probability value being greater than athreshold value, and to be a fake object in response to the secondprobability value being less than or equal to the threshold value. Forstill another example, when both the first probability value and thesecond probability value are provided as outputs by the liveness testmodel, the liveness test apparatus may determine a final probabilityvalue based on a relational equation between the first probability valueand the second probability value, and determine whether the object isfake or live by comparing the final probability value to a thresholdvalue. Here, the first probability value and the second probabilityvalues may not be in inverse relation, e.g., such as with their totalsrepresenting a 100% probability determination, but rather, the firstprobability value and the second probability value may be separatelydetermined by the liveness test model, and thus, a total of the firstprobability value and the second probability value may be representativeof less than, equal, or greater than such a 100% probabilitydetermination.

In still another example, the liveness test apparatus may determineliveness of the object at a current time based on calculations performedby the liveness test model over time, thereby representing temporalcharacteristics into the liveness determination. For example, theliveness test model may include at least one recurrent connectionstructure formed among layers of a neural network model. Through such arecurrent connection structure, an output value of one layer calculatedat the previous time may be used as an input value of that layer oranother layer at the current time, such as through feedback connections,context nodes, or short term memory or gated nodes, and thus affect anoutput at the current time. Such recurrent connections may be weightedor non-weighted. The liveness test model having such a recurrentconnection structure may be suitable for performing a liveness testtaking into consideration successive image frames. At each time theliveness test is performed on each image frame, image information of aninterest region extracted from an image frame and image information ofthe image frame of which a size is adjusted may be individually input,and internal calculations performed based on a previous image frame mayaffect the result of the liveness test for the current image frame. Insuch a case, the liveness test apparatus may perform a liveness test onan object intuitively, i.e., based on the training of the liveness testmodel, based on a movement characteristic of the object in temporallysuccessive image frames, in addition to local texture informationindicated in the interest region and global shape information indicatedin an image frame. Here, when such calculations are performed by theexample neural network, the internal calculations may be the applicationof weights to connections between nodes of different layers of theneural network, as well as operations and actuation functions of each ofthe nodes. During training, such weights and hyper-parameters of theneural network may be selectively adjusted until the liveness test modelis trained. An output layer of the neural network may be a final layerof the neural network, for example with nodes that respectively performprobability distribution operations across all the output nodes, e.g.,through softmax functions. For example, the example neural network mayhave two output nodes, the outputs of which are respectively theaforementioned first and second probability values. When properlytrained for the desired texture, global, or temporal considerations, theneural network may then intuitively perform the liveness test on theinput image data based on the trained weights of the differingconnections between the nodes within the neural network and the internalconnection and architecture of the neural network, such as with theportion of the neural network that may have the recurrent connectionstructure. In addition, the training may include performing a trainingoperation on one or more select layers of the neural networkindependently of training operations performed on other layers, eitherfor an entire training of such layers or for intermediate training ofthe layers before such layers are collectively trained with other orremaining layers, as only examples. Here, though such recurrent andoutput structures of the example liveness test neural network have beendiscussed, other layer structures of the neural network will bediscussed in greater detail further below.

FIG. 3A is a diagram illustrating an example of a method of extractingan interest region when a face image 310 is input to a liveness testapparatus. Referring to FIG. 3A, the liveness test apparatus detects aface region 320 in the face image 310. In one example, the liveness testapparatus detects landmarks in the face image 310, and detect a boundingregion including the detected landmarks as the face region 320.Alternatively, the liveness test apparatus may detect the face region320 using a Viola-Jones detector, but examples are not limited to thisspecific example. In an example, the face region 320 may be determinedso as to include substantially only the face region 320, such asexcluding background information, i.e., information beyond or other thanthe object. The liveness test apparatus then extracts interest regions,for example, eye regions 322 and 324, a nose region 326, and a lipsregion 328, from the face region 320 to perform a liveness test. Forexample, the interest regions may include the eye regions 322 and 324,the nose region 326, and the lips region 328. The liveness testapparatus may extract the interest regions 322, 324, 326, and 328 basedon locations of the landmarks, or using a visually salient regiondetecting method. In another example, the liveness test apparatusextracts the interest regions 322, 324, 326, and 328 without detectingthe face region 320 in the face image 310.

FIG. 3B is a diagram illustrating an example of a method of extractingan interest region when an eye image 330 is input to a liveness testapparatus. Here, for example, the eye image 330 may be determined so asto include substantially only an eye, eyes, or a face region, such asexcluding background information. Referring to FIG. 3B, the livenesstest apparatus detects an eye region 335 in the eye image 330. In oneexample, the liveness test apparatus detects the eye region 335 based onlandmarks detected in an eye and a region around the eye. For example,the liveness test operation may include the detection of the landmarksfrom the eye image, as well as the detection of the eye and regionaround the eye based on the detected landmarks. The liveness testapparatus then extracts, from within the eye region 335, an interestregion 345 in a form of a local patch including a pupil region 340. Forexample, the liveness test apparatus may detect the pupil region 340during the landmark detection, based on the detected landmarks, or acircular edge detector, as only examples, and determine the interestregion 345 including the detected pupil region 340. The interest region345 may be determined so as to include substantially only the pupilregion 340, for example. The pupil region 340 may include a detectedcorneal reflex 350 generated from a light source formed naturally orartificially, for example. The corneal reflex 350 and information of aregion around the corneal reflex 350 may provide a clue for determiningliveness of an object. The interest region or local patch may also berandomly selected from among plural eye or pupil regions.

FIG. 3C is a diagram illustrating an example of a method of extractingan interest region when a fingerprint image 360 is input to a livenesstest apparatus. Referring to FIG. 3C, the liveness test apparatusextracts feature points, for example, an ending point 362, a bifurcationpoint 364, a core point 366, and a delta point 368, from a ridge patternof a fingerprint in the detected fingerprint image 360. A pattern and ashape of a distribution of such feature points may provide importantinformation to determine liveness of an object. Accordingly, theliveness test apparatus may determine, to be an interest region 370, aselect region within the fingerprint pattern in which the feature pointsare densely concentrated compared to other regions of the fingerprintpattern.

FIG. 3D is a diagram illustrating an example of a method of extractingan interest region when a detected or captured vein image 380 is inputto a liveness test apparatus. Referring to FIG. 3D, the liveness testapparatus extracts a vein pattern from the vein image 380. The livenesstest apparatus extracts feature points, for example, an end point and abifurcation point of a blood vessel in the vein pattern, and determines,to be an interest region 390, a select portion of an entire region ofthe vein pattern in which the feature points are densely concentrateddistinguished from other regions of the vein pattern.

FIG. 4A is a diagram illustrating an example of a liveness test method.

Referring to FIG. 4A, a liveness test apparatus detects a global region410 in an input image, and extracts an interest region 420 in a form ofa local path from the global region 410. In iris recognition, theliveness test apparatus detects the global region 410 indicating aselect eye region in the input image, and extracts the interest region420 including a pupil region from the global region 410. For example,when a size of the global region 410 is a 200×200 pixel size, when theliveness test apparatus extracts the interest region 420 from the globalregion 410, the results may be a 64×64 pixel size image including thepupil region from the global region 410. For another example, theliveness test apparatus may extract the interest region 420 immediatelyfrom the input image without detecting the global region 410. Theliveness test apparatus may extract the interest region 420 withoutresolution adjustment, or slight resolution enhancement or down-scalingmay be performed depending on the pixel dimensions expected by the inputof the liveness test apparatus. The liveness test apparatus may alsoalternatively control the proportion of the pupil in the extractedinterest region, without resolution adjustment, so the final pixeldimensions of the extracted interest region 420 correspond to the pixeldimensions expected by the liveness test apparatus. As an example, theextracted interest region 420 may be input to the liveness testapparatus as a matrix, e.g., with pixel columns and rows correspondingto element columns and rows of the matrix, or as a three-dimensional boxwith example red, green, and blue color pixel values for each pixelbeing respectively reflected in the depth of the box.

The liveness test apparatus obtains a result of a liveness testperformed on an object from image information of the interest region 420using a trained deep convolutional neural network (DCNN) model-basedliveness test model 430. Layers of a DCNN model may be classified basedon a function of each layer, and the DCNN model may include aconvolutional layer configured to extract features through a convolutionperformed on the input image, a pooling layer configured to performabstraction to map a plurality of pixels or values from a previous layerto a lesser number of pixels or values or a single pixel or value, and afully-connected layer configured to classify features transferred from alower layer. The fully-connected or dense layer may include multiplefully-connected or dense layers. There may be multiple convolutionlayers which respectively perform convolutional filtering, for example,on connected results from a previous layer, e.g., with the convolutionallayers each outputting three-dimensional boxes whose dimensions maydepend on the filter size of the corresponding convolutional layer. Inaddition, there may be weighted connections to each convolutional layerin correspondence to each pixel of the corresponding convolutional layerand for each filter of the corresponding convolutional layer. Throughconvolution of multiple filters across the pixels in each convolutionlayer, due to the respective configurations of each convolution layer,distinguishing features of input (from the previous layer or inputlayer) image may be recognized. The DCNN may further include multiplepooling layers that may each respectively downsample input pixels orthree-dimensional boxes from a previous layer, such as withoutweighting, for example. In the example of FIG. 4A, the DCNN model-basedliveness test model 430 thus illustrates an example first convolutionlayer, then operations of a following first pooling layer arerepresented by the hashed lines, where the three-dimensional box outputfrom the first convolution layer is downsampled to a smallerthree-dimensional box, and provided to a second convolutional layer,whose output is provided to a third convolutional layer. Operations of afollowing second pooling layer are again represented by hashed lines,where the downsampled three-dimensional box is then provided to a fourthconvolutional layer, whose output is provided to a fifth convolutionallayer, and whose output is again downsampled by a third pooling layerand provided to the illustrated sixth convolutional layer. The sixthconvolutional layer is connected to the illustrated fully-connected ordense layer, which is finally connected to the illustrated two outputnode output layer. The outputs of the nodes of the illustrated outputlayer may then be provided as the illustrated test result. Throughtraining of the DCNN model-based liveness test model 430, the weightedconnections between the different layers may be set. For example, theDCNN model-based liveness test model 430 may be trained based on anumber of sample training images, e.g., with such interest regionextraction, global region extraction, and temporal application, with theweightings being adjusted through multiple iterations, such as throughbackpropagation training, until the DCNN model-based liveness test model430 accurately discerns between live and non-live input images.Accordingly, DCNN model-based liveness test model 430 may provideinformation to be used to determine liveness of the object based on theimage information input to the liveness test model 430 through internalcomputations and operations using the convolutional layer, the poolinglayer, and the fully-connected layer. Further detailed descriptions ofthe structure and function of the DCNN model are omitted here, though itis noted that alternative configurations are also available. Further,the DCNN model is provided as an illustrative example only, and theliveness test model 430 may be based on an alternate neural networkmodel of a structure other than the DCNN model. In addition, throughrepetition of at least another input layer, another first convolutionallayer, and another first pooling layer, and results thereof combinedwith results from above noted first pooling layer, further interestregions may be concurrently evaluated for a same liveness determination,such as discussed below with regard to the combination of convolutionresults for the downsized image 440 with convolution results for theinterest region in FIG. 4B.

FIG. 4B is a diagram illustrating another example of a liveness testmethod.

Referring to FIG. 4B, similarly to the example illustrated in FIG. 4A, aliveness test apparatus detects a global region 410 in an input imageand extracts an interest region 420 from the global region 410. Inaddition, the liveness test apparatus generates a downscaled image 440obtained by scaling down a size of the global region 410. For example,when the size of the global region 410 is a 200×200 pixel size, theliveness test apparatus may extract the interest region 420 of a 64×64pixel size from the global region 410 through image cropping, andgenerate the downscaled image 440 of a 64×64 pixel size from the globalregion 410 through sub-sampling. Here, the effective resolution of theinterest region 420 may be maintained, e.g., be equal to the effectiveresolution of the global region 410, while due to the downscaling ordownsampling of the global region the effective resolution of thedownscaled image 440 may be less than the effective resolution of theglobal region 410. Thus, due to the downscaling or downsampling of theglobal region, the downscaled image 440 includes less information anddetail than the original global region 410, though still demonstratingspatial information of the global region. The interest region 420 may beused to identify a delicate texture difference while reducing an amountof computation or operations, e.g., compared to an entire captured imageof a user, and the downscaled image 440 may be used to identify anoverall shape difference while reducing an amount of computation oroperations, again, compared to if such a test were performed on theentire captured image of a user. Alternatively, the liveness testapparatus may extract the interest region 420 immediately from the inputimage without detecting the global region 410, and generate thedownscaled image 440 by scaling down the input image.

Image information of the interest region 420 and image information ofthe downscaled image 440 are input to a DCNN model-based liveness testmodel 450 through different input layers of the liveness test model 450.Similar to above, the downscaled image 440 may be input to an inputlayer that connects to a seventh convolution layer, and whose output isdownsampled by a fourth pooling layer and provided to the secondconvolutional layer. The fourth pooling layer may downsample thethree-dimensional box output from the seventh convolution layer so thedownsampled three-dimensional box matches in dimensions the downsampledthree-dimensional box provided by the first pooling layer, for example.These two downsampled three-dimensional boxes may thus both haveconnections to the second convolution layer, such as so like positionedelements connect to the same nodes of the second convolution layer toimplement an element-wise summation, for example. In this example, thefirst convolutional layer may have a select convolutional structure andbe trained for extracting certain characteristic(s) of the interestregion 420, for example, and thus may have a different structure and/orbe differently trained than the seventh convolutional layer. The firstand seventh convolutional layers may also be trained together, asdescribed above using the interest region 420 and correspondingdownscaled image 440. Thus, as only an example, the convolutionalfiltering by the first convolutional layer for the interest region 420and the convolutional filtering by the seventh convolutional layer forthe downscaled image 440 may be different. Also, similar to the livenesstest model 430, each of the subsequent convolutional layers may alsohave distinct structures and objectives, for an ultimate livenessdetection determination, e.g., in combination with the remainingnon-convolutional layers of the liveness test model 450. Likewise,similar or same convolutional filters may also be included in differentconvolutional layers. Thus, the liveness test apparatus determinesliveness of an object based on an output value of the liveness testmodel 450, for example, to which the image information of the interestregion 420 and the image information of the downscaled image 440 areinput.

Alternatively, image information of the global region 410 that is notscaled down or image information of the input image may be input to theliveness test model 450 as an input, and the image information of theinterest region 420 may be input to the liveness test model 450 asanother input. As only an example, the seventh convolutional layer andfourth pooling layer may be configured for at least respective spatialfeature discrimination and sampling and for subsequent cooperation withoutputs of the first pooling layer after the first convolutional layerthat discriminates textural information of the interest region 420. Theliveness test model 450 may output a result of a test to determine theliveness of the object based on the input image information.

FIG. 5 is a diagram illustrating still another example of a livenesstest method.

A liveness test apparatus may perform a liveness test on temporallysuccessive image frames. The liveness test apparatus may thus perform aliveness test on an object based further on a movement characteristic ofthe object based on a lapse of time that is indicated in the imageframes, in addition to a texture characteristic and a shapecharacteristic indicated in each image frame. The liveness testapparatus may detect liveness of the object more accurately bydetermining the liveness based on a temporal characteristic such as themovement characteristic of the object, in addition to a spatialcharacteristic of an image frame such as the texture characteristic andthe shape characteristic.

Referring to FIG. 5, in stage 512, the liveness test apparatus receivesa first image frame. The liveness test apparatus detects a global regionin stage 514, and extracts an interest region from the global region instage 516. In stage 540, the liveness test apparatus performs a livenesstest on an object in the first image frame using a liveness test model.For example, image information of the interest region and imageinformation of a downscaled image obtained by scaling down the globalregion may be input to the liveness test model.

Subsequently, a second image frame, which is a subsequent image frame ofthe first image frame, is input to the liveness test apparatus, and theliveness test apparatus performs a liveness test on an object in thesecond image frame. In stages 522 through 526, the liveness testapparatus receives the second image frame and detects a global regionfrom the second image frame, and extracts an interest region from thedetected global region. In stage 540, the liveness test apparatusperforms the liveness test on the object in the second image frame basedon calculations performed by the liveness test model from the firstimage frame, in addition to the image information of the interest regionand the image information of the downscaled image obtained by scalingdown the global region. The liveness test apparatus repeats the stagesdescribed in the foregoing for each subsequent image frame to perform aliveness test on an N-th image frame through stages 532, 534, 536, and540.

The liveness test model may implement this temporal consideration usingthe aforementioned recurrent connection structure where the livenesstest model operates as having memory or context of the previousoperations and previous images for subsequent operations and subsequentimages. For example, with such a recurrent connection structure,calculated values of the liveness test model derived in a previousliveness test operation performed on a previous image frame may affect aliveness test to be performed on a current image frame. As noted above,in a neural network example, the recurrent structure may be representedby outputs of nodes of one layer being connected back to the same layer,being fed back to a previous layer, provided to a context layer or nodefor subsequent output, or implemented by nodes having short term orgated memories, as only example. The recurrent connection structure ofthe liveness test model may vary based on examples. For example, basedon a shape of the recurrent connection structure, a result of theliveness test to be performed on the current image frame may be affectedonly by a result of the liveness test performed on one previous imageframe, a select number of previous image frames, or by a result ofliveness tests performed on all image frames presented at or since aprevious time.

FIG. 6 is a diagram illustrating an example of the liveness test methoddescribed with reference to FIG. 5.

Referring to FIG. 6, similarly to the example illustrated in FIG. 4B, aliveness test apparatus detects a global region 410 in an input image,for example, an image frame, and extracts an interest region 420 fromthe global region 410. In addition, the liveness test apparatusgenerates a downscaled image 440 obtained by scaling down a size of theglobal region 410. Alternatively, the liveness test apparatus extractsthe interest region 420 immediately from the input image withoutdetecting the global region 410, and generates the downscaled image 440by scaling down the input image.

Image information of the interest region 420 and image information ofthe downscaled image 440 may be input to a liveness test model 610through different input layers of the liveness test model 610. Theliveness test apparatus obtains a result of a liveness test performed onan object from the image information of the interest region 420 and theimage information of the downscaled image 440 using the liveness testmodel 610. Thus, in addition to the configurations of FIGS. 4A and 4B,the example fully-connected or dense layer of FIG. 6 may includemultiple fully-connected or dense layers that include the above examplerecurrent structures, as well as two output layers. The liveness testmodel 610 may have such a recurrent connection structure in which aprevious calculated values of the liveness test model 610 with respectto interest region 420 and downscaled image 440 of previous frames areinput to a layer of the liveness test model 610 that is implementing theliveness test with respect to the interest region 420 and downscaledimage 440 of a current frame at a current time. Accordingly, theliveness test model 610, which is provided in a neural networkstructure, may include one or more layers configured to extract atexture features of each image frame, one or more layers configured toextract spatial features of each image frame, and one or more layersconfigured to extract temporal features of the object that changes overtime, and perform a liveness test based on all of the texture, spatial,and temporal features obtained by the corresponding layers.

FIG. 7 is a diagram illustrating an example of a liveness test apparatus700. The liveness test apparatus 700 may perform a liveness test on anobject in an input image, and output a result of the liveness test.Referring to FIG. 7, the liveness test apparatus 700 includes aprocessor 710 and a memory 720.

The processor 710 performs one or more or all operations or stagesdescribed with reference to FIGS. 1A through 6. For example, theprocessor 710 may extract an interest region from the input image, andperform the liveness test on the object in the input image using aliveness test model that uses image information of the interest regionas a first input. According to an example, the liveness test model mayuse, as a second input, image information of a global region or adownscaled image of the global image, in addition to the imageinformation of the interest region. The liveness test model may have arecurrent connection structure that is affected by a result of previouscalculations for previous frames.

When the object in the input image is determined to be a fake object,the processor 710 may generate a signal corresponding to a failure inthe liveness test. In such a case, recognition or authentication of theobject may be immediately determined to be unsuccessful, and theprocessor 710 may cease authentication operations, and thus notimplement further recognition or authentication on the object, such asfull facial recognition of the user. Conversely, when the object in theinput image is determined to be a live object, the processor 710 maygenerate a signal corresponding to a success in the liveness test. Insuch a case, the processor 710 may implement further recognition orauthentication on the object, such as by using the previously capturedimage(s) for facial recognition or a new image may be captured.Alternatively, if the object in the input image is determined to be alive object, the processor 710 may not implement further recognition orauthentication and merely enable the user access to further functions ofthe processor, enable the user to obtain access to a restricted orcontrolled area, or enable the user access to financial services orauthorized payment, such as through a mobile payment system representedby the processor 710. As noted, when liveness of the object isdetermined, a series of stages for further recognition or authenticationmay be performed on the object. However, when the liveness of the objectis not determined, such further recognition or authentication may not beperformed on the object, and all the stages may be terminated. Inaddition, the processor 710 may perform training of a liveness testmodel, such as the above discussed DCNN model-based liveness test model.For example, the training may be performed through backpropagationtraining the DCNN model-based liveness test model based on capturingsuch interest regions from input images or determined global images ofsuch input images from labeled training data images, for example. Thetraining data may include images, e.g., live images, of a user of theliveness test apparatus 700 and/or corresponding presorted/labeledtraining images from a general collection of people and/or select groupsof people, as only non-limiting examples.

The processor 710 may be embodied as an array of a plurality of logicgates, or have other hardware processing configurations, depending onembodiment. As only examples, the processor 710 may include multipleprocessor cores or be representative of a multi-processor system ofcluster, or the processor 710 may be a multi-core graphics processorunit (GPU), again noting that alternatives are available.

The memory 720 stores instructions to perform one or more or alloperations or stages described with reference to FIGS. 1A through 6,and/or stores results of calculations performed by the liveness testapparatus 700. The memory 720 may include a non-transitorycomputer-readable medium, for example, a high-speed random access memory(RAM) and/or a nonvolatile computer-readable storage medium (forexample, at least one disk storage device, flash memory device, or othernonvolatile solid-state memory device). In addition, the memory 720 maystore parameters of the liveness test model, such as pre-trained weightsfor a DCNN model-based liveness test model, as discussed above.

FIG. 8 is a diagram illustrating an example of a computing apparatus800.

The computing apparatus 800 may perform a liveness test on an object inan input image, and perform user authentication. The computing apparatus800 may include a function of the liveness test apparatus 700 describedwith reference to FIG. 7. Referring to FIG. 8, the computing apparatus800 includes a processor 810, a memory 820, a camera 830, a storagedevice 840, an input device 850, an output device 860, and a networkinterface 870, for example.

The processor 810 performs a function and executes instructions tooperate in the computing apparatus 800. For example, the processor 810processes instructions stored in the memory 820 or the storage device840. The processor 810 is configured to perform one or more or alloperations or stages described above with respect to FIGS. 1A through 6.In addition, the processor 810 is configured to control other functionsof the computing apparatus 800. For example, the computing apparatus maybe mobile device, such as a mobile phone, tablet, or personal computer,and thus the processor 810 is further configured to implement othertypical functions of the computing apparatus 800.

The memory 820 stores information in the computing apparatus 800. Thememory 820 includes a computer-readable storage medium or acomputer-readable storage device. The memory 820 may include, forexample, a RAM, a dynamic RAM (DRAM), a static RAM (SRAM), and othertypes of a nonvolatile memory known in the art to which the examplesdescribed herein belong. In addition, memory 820 is furtherrepresentative of multiple such types of memory. The memory 820 alsostores the instructions to be executed by the processor 810 and storesrelated information during operations of software or an applicationperformed by the computing apparatus 800.

The camera 830 obtains a still image, a video image, or both of theimages. In an example, the camera 830 captures a face region or an irisregion that is input by a user for user authentication. In addition, asnoted above, the camera 830 may also be controlled by the processor 810during other functions of the computing apparatus 800, such as whenoperated as a personal camera.

The storage device 840 includes a computer-readable storage medium or acomputer-readable storage device. The storage device 840 stores agreater amount of information than the memory 820, and stores theinformation for a long period of time. The storage device 840 includes,for example, a magnetic hard disk, an optical disk, a flash memory, andan electrically erasable programmable read-only memory (EPROM), a floppydisk, and other types of a nonvolatile memory known in the art to whichthe examples described herein belong.

The input device 850 receives an input from the user, for example, atactile input, a video input, an audio input, and a touch input. Theinput device 850 detects the input from, for example, a keyboard, amouse, a touchscreen, a microphone, a fingerprint reader, a retinascanner, and the user, and includes another device configured totransfer the detected input to the computing apparatus 800.

The output device 860 provides the user with an output of the computingapparatus 800 through a visual, audio, or tactile channel. The outputdevice 860 includes, for example, a liquid crystal display (LCD), alight-emitting diode (LED) display, a touchscreen, a speaker, avibration generator, or another device configured to provide the userwith the output.

The network interface 870 communicates with an external device through awired or wireless network. The network interface 870 includes, forexample, an Ethernet card, an optical transceiver, a radio frequencytransceiver, or another network interface card configured to transmit orreceive information. The network interface 870 communicates with anexternal device using Bluetooth, WiFi, or a third generation (3G),fourth generation (4G), or fifth generation (5G) communication method.The network interface 870 may further include a near field transceiveror the like. For example, through control of the processor 810 uponauthentication of a user, the near field transceiver may transmit apayment authorization to an external terminal, such as with anappropriate mobile payment instruction transmitted by the near fieldtransceiver. In addition, the processor 810 may control the networkinterface 870 to routinely check for updates for the liveness testmodel, and request, receive, and store parameters or coefficients of thesame in the memory 820. For example, when the liveness test model is theabove example DCNN model-based liveness test model, the processor 810may request, receive, and store updated weighting matrices for the DCNNmodel-based liveness test model. In addition, updated hyper-parametersthat can control or alter the configuration or architecture of the DCNNmodel-based liveness test model may also be requested, received, andstored along with corresponding weighting matrices.

The computing apparatus 120, camera 130, display 125, home button 170,computing apparatus 180, liveness test apparatus, processor 710, memory720, computing apparatus 800, liveness test apparatus 700, processor810, memory 820, camera 830, storage device 840, input device 850,output device 860, and network device 870 in FIGS. 1A-8 and that performthe operations described in this application are implemented by hardwarecomponents configured to perform the operations described in thisapplication that are performed by the hardware components. Examples ofhardware components that may be used to perform the operations describedin this application where appropriate include controllers, sensors,generators, drivers, memories, comparators, arithmetic logic units,adders, subtractors, multipliers, dividers, integrators, and any otherelectronic components configured to perform the operations described inthis application. In other examples, one or more of the hardwarecomponents that perform the operations described in this application areimplemented by computing hardware, for example, by one or moreprocessors or computers. A processor or computer may be implemented byone or more processing elements, such as an array of logic gates, acontroller and an arithmetic logic unit, a digital signal processor, amicrocomputer, a programmable logic controller, a field-programmablegate array, a programmable logic array, a microprocessor, or any otherdevice or combination of devices that is configured to respond to andexecute instructions in a defined manner to achieve a desired result. Inone example, a processor or computer includes, or is connected to, oneor more memories storing instructions or software that are executed bythe processor or computer. Hardware components implemented by aprocessor or computer may execute instructions or software, such as anoperating system (OS) and one or more software applications that run onthe OS, to perform the operations described in this application. Thehardware components may also access, manipulate, process, create, andstore data in response to execution of the instructions or software. Forsimplicity, the singular term “processor” or “computer” may be used inthe description of the examples described in this application, but inother examples multiple processors or computers may be used, or aprocessor or computer may include multiple processing elements, ormultiple types of processing elements, or both. For example, a singlehardware component or two or more hardware components may be implementedby a single processor, or two or more processors, or a processor and acontroller. One or more hardware components may be implemented by one ormore processors, or a processor and a controller, and one or more otherhardware components may be implemented by one or more other processors,or another processor and another controller. One or more processors, ora processor and a controller, may implement a single hardware component,or two or more hardware components. A hardware component may have anyone or more of different processing configurations, examples of whichinclude a single processor, independent processors, parallel processors,single-instruction single-data (SISD) multiprocessing,single-instruction multiple-data (SIMD) multiprocessing,multiple-instruction single-data (MISD) multiprocessing, andmultiple-instruction multiple-data (MIMD) multiprocessing.

The methods illustrated in FIG. 1A-8 that perform the operationsdescribed in this application are performed by computing hardware, forexample, by one or more processors or computers, implemented asdescribed above executing instructions or software to perform theoperations described in this application that are performed by themethods. For example, a single operation or two or more operations maybe performed by a single processor, or two or more processors, or aprocessor and a controller. One or more operations may be performed byone or more processors, or a processor and a controller, and one or moreother operations may be performed by one or more other processors, oranother processor and another controller. One or more processors, or aprocessor and a controller, may perform a single operation, or two ormore operations.

Instructions or software to control computing hardware, for example, oneor more processors or computers, to implement the hardware componentsand perform the methods as described above may be written as computerprograms, code segments, instructions or any combination thereof, forindividually or collectively instructing or configuring the one or moreprocessors or computers to operate as a machine or special-purposecomputer to perform the operations that are performed by the hardwarecomponents and the methods as described above. In one example, theinstructions or software include machine code that is directly executedby the one or more processors or computers, such as machine codeproduced by a compiler. In another example, the instructions or softwareincludes higher-level code that is executed by the one or moreprocessors or computer using an interpreter. The instructions orsoftware may be written using any programming language based on theblock diagrams and the flow charts illustrated in the drawings and thecorresponding descriptions in the specification, which disclosealgorithms for performing the operations that are performed by thehardware components and the methods as described above.

The instructions or software to control computing hardware, for example,one or more processors or computers, to implement the hardwarecomponents and perform the methods as described above, and anyassociated data, data files, and data structures, may be recorded,stored, or fixed in or on one or more non-transitory computer-readablestorage media. Examples of a non-transitory computer-readable storagemedium include read-only memory (ROM), random-access memory (RAM), flashmemory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs,DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, magnetictapes, floppy disks, magneto-optical data storage devices, optical datastorage devices, hard disks, solid-state disks, and any other devicethat is configured to store the instructions or software and anyassociated data, data files, and data structures in a non-transitorymanner and provide the instructions or software and any associated data,data files, and data structures to one or more processors or computersso that the one or more processors or computers can execute theinstructions. In one example, the instructions or software and anyassociated data, data files, and data structures are distributed overnetwork-coupled computer systems so that the instructions and softwareand any associated data, data files, and data structures are stored,accessed, and executed in a distributed fashion by the one or moreprocessors or computers.

While this disclosure includes specific examples, it will be apparentafter an understanding of the disclosure of this application thatvarious changes in form and details may be made in these exampleswithout departing from the spirit and scope of the claims and theirequivalents. The examples described herein are to be considered in adescriptive sense only, and not for purposes of limitation. Descriptionsof features or aspects in each example are to be considered as beingapplicable to similar features or aspects in other examples. Suitableresults may be achieved if the described techniques are performed in adifferent order, and/or if components in a described system,architecture, device, or circuit are combined in a different manner,and/or replaced or supplemented by other components or theirequivalents. Therefore, the scope of the disclosure is defined not bythe detailed description, but by the claims and their equivalents, andall variations within the scope of the claims and their equivalents areto be construed as being included in the disclosure.

1. A processor implemented liveness test method, the liveness testmethod comprising: extracting an interest region of an object from aportion of the object in an input image; performing a liveness test onthe object using a neural network model-based liveness test model foreach of one or more images of the object respectively input to theliveness test model, including the liveness test model using imageinformation of the interest region as provided first input imageinformation to the liveness test model and determining liveness of theobject based at least on extracted texture information from the imageinformation of the interest region by the liveness test model; andindicating a result of the liveness test.
 2. A processor implementedliveness test method, the liveness test method comprising: extracting aninterest region of an object from a portion of the object in an inputimage; performing a liveness test on the object using a neural networkmodel-based liveness test model, the liveness test model using imageinformation of the interest region as provided first input imageinformation to the liveness test model and determining liveness based atleast on extracted texture information from the image information of theinterest region by the liveness test model; and indicating a result ofthe liveness test, wherein the liveness test model uses imageinformation of the input image or image information of the portion ofthe object in the input image as provided second input image informationto the liveness test model that is also provided the first input imageinformation, wherein the determining of the liveness is based at leaston the extracted texture information from the first input imageinformation and spatial information extracted based on the second inputimage information by the liveness test model, and wherein an effectiveresolution of the first input image information is higher than aneffective resolution of the second input image information.
 3. Themethod of claim 2, further comprising: adjusting a pixel size of atleast the image information of the portion of the object in the inputimage, after the extracting of the interest region, to generate thesecond input image information.
 4. (canceled)
 5. The method of claim 3,wherein a pixel size of the first input image information is equal to apixel size of the second input image information.
 6. The method of claim2, wherein the liveness test model includes a recurrent connectionstructure that is configured so a determination of liveness of theobject, with respect to the input image being a current frame, isdependent on results of the liveness test model obtained with respect toa previous frame.
 7. The method of claim 2, wherein the first inputimage information and the second input image information are input tothe liveness test model through different input layers of the livenesstest model.
 8. The method of claim 7, wherein the liveness test modelindependently performs filtering on the first input image informationand the second input image information before performing furtherfiltering on a combination of information of results from both theperformed filtering on the first input image information and theperformed filtering on the second input image information.
 9. The methodof claim 1, wherein the extracting of the interest region comprises:extracting a select portion including a pupil region as the interestregion from the portion of the object in the input image.
 10. The methodof claim 9, wherein the liveness test model uses image information ofthe portion of the object in the input image, within a facial region ofthe object and including the select portion, as provided second inputimage information to the liveness test model that is also provided thefirst input image information, and wherein the determining of theliveness is based at least on the extracted texture information from theselect portion and spatial information extracted based on the secondinput image information by the liveness test model.
 11. The method ofclaim 1, wherein the extracting of the interest region comprises:extracting respective select portions, from the portion of the object inthe input image, including at least one of an eye, a nose, and lips asthe interest region.
 12. The method of claim 11, wherein the livenesstest model uses image information of the portion of the object in theinput image, within a facial region of the object and including one ormore of the respective select portions, as provided second input imageinformation to the liveness test model that is also provided the firstinput image information, and wherein the determining of the liveness isbased at least on extracted texture information from the one or more ofthe respective select portions by the liveness test model and spatialinformation extracted based on the second input image information by theliveness test model.
 13. The method of claim 1, wherein the extractingof the interest region comprises: extracting respective one or moreselect portions, from a fingerprint region as the portion of the objectin the input image, as respective interest regions.
 14. The method ofclaim 13, wherein the liveness test model uses image information of theportion of the object in the input image, as provided second input imageinformation to the liveness test model that is also provided the firstinput image information, and wherein the determining of the liveness isbased at least on extracted texture information from the respective oneor more select portions by the liveness test model and spatialinformation extracted based on the second input image information by theliveness test model.
 15. The method of claim 1, wherein the extractingof the interest region comprises: extracting a select portion, from avein region as the portion of the object in the input image, as theinterest region.
 16. The method of claim 15, wherein the liveness testmodel uses image information of the object in the portion of the inputimage, as provided second input image information to the liveness testmodel that is also provided the first input image information, andwherein the determining of the liveness is based at least on theextracted texture information from the select portion and spatialinformation extracted based on the second input image information by theliveness test model.
 17. The method of claim 1, wherein the extractingof the interest region comprises: detecting feature points in the inputimage to determine the portion of the object in the input image; anddetermining the interest region based on the detected feature points.18. The method of claim 1, wherein a location of the interest region isdetermined based on a determined type of biometric information for whichthe liveness test is determined to be performed.
 19. The method of claim1, further comprising selectively controlling access to functions of acomputing apparatus based on results of the performed liveness test. 20.A non-transitory computer-readable storage medium storing instructions,that when executed by a processor, cause the processor to perform themethod of claim
 1. 21. Aliveness test computing apparatus comprising: atleast one processor configured to: extract an interest region of anobject from a portion of the object in an input image; and perform aliveness test on the object using the neural network model-basedliveness test model for each of one or more images of the objectrespectively input to the liveness test model, including the livenesstest model using provided image information of the interest region asfirst input image information to the liveness test model and determiningliveness of the object based at least on extracted texture informationfrom the image information of the interest region by the liveness testmodel.
 22. The liveness test computing apparatus of claim 21, whereinthe liveness test model is configured to use image information of theinput image or image information of the portion of the object in theinput image as provided second input image information to the livenesstest model that is also provided the first input image information, witha pixel size of at least the image information of the portion of theobject in the input image being adjusted after the extracting of theinterest region to generate the second input image information, andconfigured to provide a reference value indicating a result of theliveness determination, and wherein the determining of the liveness isbased at least on the extracted texture information from the first inputimage information and spatial information extracted based on the secondinput image information by the liveness test model.
 23. The livenesstest computing apparatus of claim 22, wherein the liveness testcomputing apparatus is further configured to input the first input imageinformation and the second input image information to the liveness testmodel through different configured input layers of the liveness testmodel.
 24. The liveness test computing apparatus of claim 23, whereinthe liveness test model is configured to independently perform filteringon the first input image information for extracting the textureinformation and filtering on the second input image information forextracting the spatial information before performing further filteringon a combination of information of results from both the performedfiltering on the first input image information and performed filteringon the second input image information.
 25. The liveness test computingapparatus of claim 21, wherein the liveness test model includes arecurrent connection structure that is configured so a determination ofliveness of the object, with respect to the input image being a currentframe, is dependent on results of the liveness test model obtained withrespect to a previous frame.
 26. The liveness test computing apparatusof claim 21, wherein the processor is further configured to selectivelycontrolling access to functions of the liveness test computing apparatusbased on results of the performed liveness test.
 27. A liveness testcomputing apparatus comprising: at least one processor configured to:extract an interest region of an object within a detected face region inan input image; perform a liveness test on the object using aconvolutional neural network by providing a first input layer of theconvolutional neural network image information of the interest region asfirst input image information, providing a second input layer of theconvolutional neural network second input image informationcorresponding to the detected face region, and determining livenessbased on a combination of first results dependent on texture informationextracted from the image information of the interest region through afirst convolutional layer of the convolutional neural network and secondresults dependent on spatial information extracted from the second inputimage information through a second convolutional layer of theconvolutional neural network independent of the first convolutionallayer; and selectively control access to other functions of the livenesstest computing apparatus based on results of the performed livenesstest.
 28. The liveness test computing apparatus of claim 27, wherein theprocessor is configured to provide the second input layer of theconvolutional neural network the second input image informationcorresponding to the detected face region, with a pixel size of at leastimage information of the face region being adjusted after the extractingof the interest region to generate the second input image information.29. The liveness test computing apparatus of claim 27, wherein theconvolutional neural network further includes a third convolutionallayer configured to perform filtering on the combination of firstresults and the second results.
 30. The method of claim 1, wherein theliveness test model uses image information of the input image or imageinformation of the portion of the object in the input image as providedsecond input image information to the liveness test model that is alsoprovided the first input image information, and wherein the determiningof the liveness is based at least on the extracted texture informationfrom the first input image information and spatial information extractedbased on the second input image information by the liveness test model.31. The method of claim 31, further comprising: adjusting a pixel sizeof at least the image information of the portion of the object in theinput image, after the extracting of the interest region, to generatethe second input image information having a pixel size same as a pixelsize of the image information of the interest region.